Autonomous Cryogenics Loading Operations Simulation 
Software: Knowledgebase Autonomous Test Engineer 

Walter S. Wehner Jr. 

KENNEDY SPACE CENTER 

Major: Computer Science 

Program KSC FO Summer Session 

Date: 12 08 2013 



NASA USRP - Internship Final Report 


Autonomous Cryogenics Loading Operations Simulation 
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Walter S. Wehner, Jr. 1 

New Jersey Institute of Technology, Newark, New Jersey, 07102 


Working on the ACLO (Autonomous Cryogenics Loading Operations) project I have had 
the opportunity to add functionality to the physics simulation software known as KATE 
(Knowledgebase Autonomous Test Engineer), create a new application allowing WYSIWYG 
(what-you-see-is-what-you-get) creation of KATE schematic files and begin a preliminary 
design and implementation of a new subsystem that will provide vision services on the IHM 
(Integrated Health Management) bus. The functionality I added to KATE over the past few 
months includes a dynamic visual representation of the fluid height in a pipe based on 
number of gallons of fluid in the pipe and implementing the IHM bus connection within 
KATE. I also fixed a broken feature in the system called the Browser Display, implemented 
many bug fixes and made changes to the GUI (Graphical User Interface). 


Nomenclature 

s = arc length 

h = fluid height 

c = cord length 

d = sector height 

R = pipe radius 

0 = sector angle 

I. Introduction 

K ATE (Knowledgebase Autonomous Test Engineer) is a software system that uses a physics model of an 
environment to monitor and identify suspected component failures. The environments are defined by a 
knowledgebase approach which allows for the abstraction or rules and behaviors from the environment. This level of 
abstraction gives the knowledgebase approach to modeling environments power and flexibility not available in an 
expert system. The physics model is implemented in KATE by creating reusable objects. These objects are then 
consumable in a new knowledgebase. This approach of implementing reusable objects allows a new environment to 
be modeled starting with all the previous objects. 

Working on the KATE codebase I added a GUI (Graphical User Interface) object to KATE that represents the 
amount of fluid in a horizontal pipe, fixed a broken feature known as the CUI (Compact Unique Identifier) Browser 
and implemented the connection to the IHM (Infrastructure Health Management) bus. In support of KATE I created 
an application that allows a schematic to be designed in a WYSIWIG (what-you-see-is-what-you-get) style editor. I 
also began the design and implementation of a new service that will be available on the IHM bus. This service will 
provide image analysis and computer vision support to the ACLO (Autonomous Cryogenics Loading Operations) 
project and the other subsystems on the IHM bus. 

The remainder of this paper is organized as follows. Section II will describe the changes and additions to KATE. 
Section three will describe the xyFile format, the class architecture and the implementation of the Schematic 
Designer. Section IV will describe the application architecture the class design for camera control, the class design 
for image analysis and the GUI implementation of the Visual Information Service and Section V will provide 
conclusions. 


1 Computer Science Masters Student, Computer Science, New Jersey Institute of Technology. 
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II. KATE Changes and Additions 
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The initial KATE project was started in the nineteen nineties as a port from a LISP version of the software to a 
C++ version. My first few changes to KATE gave me the opportunity to get familiar with the one-hundred plus file 
codebase and the coding style used by the team that implemented the port. My first changes include added coloring 
to different display elements so 
you could easily differentiate 
between modeled values and 
physical value; I cleaned up 
some compiler warnings and 
fixed some memory leaks that 
where reported running a tool 
named valgrind; and I fixed 
some broken code that caused 
the CUI Browser to no longer 
be displayed. Figure one shows 
the CUI browser and the 
coloring changes implemented 
where white backgrounds 
denote physical elements and blue backgrounds denote modeled elements. 

After familiarizing myself with the codebase and how the system worked internally, I implemented a new 

feature known as the Fluid Height Gauge control. This new control 
displays a graphical representation of the fluid height in a horizontal 
pipe. The requirements for implementing the new control were that it 
needed to be scalable and the only inputs supplied are pipe diameter, 
length and number of gallons in the pipe. The control needed to 
subscribe to updates from the system to provide real-time updates to 
the number of gallons in a pipe and update its display. I inherited 
from the class “UpdateObject” and implemented the necessary 
functions for calculating the fluid height and creating the graphical 
representation of the pipe cross-section. 

Knowing the length of a pipe and the number of gallons in the 


Figure 1. CUI Browser. 



Figure 2. Circle segment and target area. 


pipe would allow me to solve for 
the diameter of a pipe that would 
be completely full. Knowing this 
diameter would then allow me to 
calculate the cross-sectional area 
of the volume. This is the target 
arc area in the original pipe that is 
full represented in figure 2 as c. 
Once we calculated the target area 
to fill in the original pipe we are 
left with two unknowns in 
calculating the arc area, the height, 
h, and an angle theta, 0. 

In figure 3, we present the 
algorithm for calculating the arc 
area using an iterative approach 
incrementing theta and calculating 
the arc area to find the fluid 
height. 

The process is the same for 
finding the height above half full 
but instead of solving for the 
remaining full we solve for the 
amount empty and subtract the 


///////////////////////////////////////////////////////////// 

// UpdateFieldFluidHeightGauge: :CalculateTheta 
// 

// Calculate the angle needed to draw the arc that displays the fluid cr empty section 
// inputs: radius • the radius cf the pLpe 

// targetArcArea the area under the sector that would be full or empty 

// Outputs: returns the angle that best fits the area of the sector 

double UpdateFieldFi uidHeightGauce: :Cal.culateTheta(dof/jbIe radius, double targetArcArea) 
{ 

doubl increment = . ; 

double theta = 0 ; 

double cordLength =3 ; 

doubl arc Area = .3; 

doubl ad]ustmentTriangle = . ; 

double sectorArea = 3; 

doubl beta - ■; 


(arc Area <= targetArcArea) 

{ 

theta += increment; 

cordLength =2. * radius * .m (.5 * theta); 

beta = cos ( . 5 * theta} *■ radius; 
adjustmentTnangle = .' * cordLength * beta; 
sectorArea = ((.5 * theta) * ( radius ♦radius) ) ; 
arcArea = sectorArea - adjustmentTriangle; 

} 

(theta - increment); 

} 


Figure 3. Algorithm for calculating theta. 
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calculated height from the diameter of the pipe. The process of drawing the control also uses the calculated values of 
theta and fluid height. We convert the values based on the control size into pixels per inch to maintain scaling of 
different sized controls and various sized pipe segment. In figure 4, we show two different fluid height controls 
being displayed on a schematic view within KATE. 

One requirement for KATE was to be able to 
communicate with the other systems in the ACLO 
project. The communication between different 
systems used an interface called ICE (Internet 
Communications Engine). I worked with Kelvin 
Ruiz to integrate the ICE code and implement the 
necessary changes in KATE for connecting to the 
1HM (Infrastructure Health Management) bus. 

Kelvin was designing the GUI interface for what is 
known to KATE as a Data Provider. The IHM bus 
was a generic Data Provider class that would take 
messages and convert them to a KATE friendly 
structure for consumption within KATE. It also 
takes messages from KATE and publish these 
messages for other systems to consume and act 
upon. I worked on understanding the changes 
needed for the communications and implementing 
those changes in KATE while Kelvin completed 
the GUI. Once Kelvin completed the GUI I worked with him to implement my changes in the new generic data 
provider module. 


TK001 


MV150 


rwwol 


Figure 4. Fluid Height Gauge GUI Control. 



III. Schematic xyFile Designer 

KATE has a feature that allows for an image to be loaded and then a set of CUI’s with values be displayed on 
top of the image. This feature is known as the Schematic View within the KATE application. The current method of 
developing these files is by opening your favorite text editor and hand coding all the values. The Schematic xyFile 
Designer application developed is a utility program created for windows that allows for the generation of “schematic 
xy files” for use in KATE. This application allows for the WYSIWYG creation of the xy files so a user does not 
need to know the internal structure to create the file or hand code the file in a text editor. 



Figure 5. KATE Schematic xy File Designer. 
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A. KATE xy File Format 

The KATE xy file has a specific structure that allows for the loading of the graphical objects to be displayed on 
the schematic view. The encoding of the file is ASCII and the newline character is ‘\n’ as opposed to ‘\r\n’ which is 
standard in windows. The first line of the file is XYFILE followed by a new line. After the first line, the rest of the 
file is a listing of the elements to be displayed which are separated by a newline. The element information block 
consists of up to fifteen parameters depending on the class type being displayed. The first parameter is the index 
value. The second parameter is the class type value and will determine if we load additional values. 


Value Display Class (classtype = 0) 

Fluid Height Gauge (classtype = 1) 

Tank Fluid Height (classtype = 2) 

name (text) 

pipe diameter (floating point) 

tank top height (floating point) 

xloc (integer) 

pipe length (floating point) 

tank mid height (floating point) 

yloc (integer) 

control width (integer) 

tank bottom height (floating point) 

xl (integer) 

control height (integer) 

semi major axis (floating point) 

x2 (interger) 

name (text) 

semi minor axis (floating point) 

show (integer) 

x loc (integer) 
y loc (integer) 
xl (integer) 
x2 (interger) 
show (integer) 

control width (integer) 
control height (integer) 
name (text) 
x_loc (integer) 
y loc (integer) 
xl (integer) 
x2 (interger) 
show (integer) 

Table 1. xy File types and associated parameters. 



Table 1 shows the additional parameters for each classtype loaded. All parameters are delimited by a new line 
character. 


B. Class Design 

The class structure implemented begins with an abstract class 
called “Knowledgebaseltem” This abstract class in inherited by the 
“ValueDisplayltem”, “FluidHeightGaugeltem” and 

“TankFluidHeightltem” classes. These classes hold the data that is 
written to the file and implement the interface to alert the 
framework when a property value has change. To hold these items 
and use them we created a classes called “KnowledgebaseltemList” 
which extends the “ObservableCollection” framework class. This 
list will notify the framework when there are changes to the 
collection such as adding and removing elements. 

We then must implement the controls to display our data. We 
start by designing a class called “Knowledgebaseltem Visual” 
which will be exteneded byt three children called 

“ValueDisplayltemVisual”, “FluidHeightGaugeVisual” and 
“TankFluidHeightVisual”. These classes handle drawing an 
individual knowledgebase item on the screen. Next, we create a 
control called the “KnowledgebaseltemListVisualizer”. This class 
extends the class “Frame workElement” and implements listening to 
changes in the “KnowledgebaseltemList”, drawing the 

“Knowledgebasedltem Visual” classes. 


C. Implementation 

The implementation of our application is done by 
creating a user control called “SchematicDesigner”. The 
“SchematicDesigner” class which inherits from the class 
“UserControl”. The “SchematicDesigner” is a control that handles 
adding and removing “KnowledgebaseltemVisual”, displaying a 
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j .position 

.visible 
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Figure 6. Knovvledgebaseltem Class. 
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schematic image background and the 
dragging and dropping of the items into place. 
In the main application window we use 

this control and a ListBox control that 
loads the Knowledgebaseltems. We are able 
to drag the CUI name from the ListBox and 
move them in place on the SchematicDesigner 
control. The mail window class handles 
loading a “KATE flatfile”, loading a “Kate xy 
File” and saving a “KATE xy file”. 

We show the WYSIWYG Schematic 
Designer application in Figure 5. The 
application is displaying a Fluid Height Gauge 
and multiple Display Value items over a 
background image and a list of CUTs that are 
able to be added to the schematic. 

Please see the Appendix for a review of all 
class diagrams used in the KATE Schematic 
xyFile Designer. 


Figure 7. SchematicDesigner Class. 
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IV. Visual Information Services Application 

The VIS (Visual Information Services) is an application that will add computer vision and image analysis 
support to the ACLO project. The requirements for VIS are that it be a windows based application but the 
application logic be developed in C++ and be portable to Linux; and the application will be available for other 
systems on the IHM bus. 

D. Application Architecture 

The Windows GUI is written in C# using WPF (Windows Presentation Foundation). C# is a managed language 
and uses the .NET Framework. The application code is packaged in a DLL (Dynamic Link Library) and is written in 
native C++. To use the native DLL in the .NET Framework I needed to implement a managed C++ wrapper to 
handle creating and disposing of the native objects and converting between native data types and managed data 
types. There are many benefits to this design including portability between platforms; flexibility in GUI design and 
application implementation; and reusability of image processing and computer vision functions. 

I have created a DLL called “CameraControl.dll” that is responsible for communications between the application 
and the network cameras in the Cryogenics Test Bed. This DLL provides functions to move a camera, retrieve 
images from a camera and query/set some camera parameters. This DLL is written in native C++ and is wrapped by 
a DLL called “ApplicationBridge.dll”. I expose a few underlying classes in this application wrapper and handle all 
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memory management for the native object creation. The 
“ApplicationBridge.dll” is then consumed by the GUI written in 
C# and I use the GUI to load a database of CUI locations, request 
images and perform image processing and analysis. 

The image processing and analysis routines are in another 
DLL which is currently written in C#. The goal will be to write 
this DLL in native C++ to make it portable between platforms 
and consumable by any client application. In the image 
processing DLL we have implemented functions for gray scale 
conversion, binary conversion, skeletonization, edge detection 
using Sobel filtering, image smoothing and circle detection by 
Edge Oriented Hough Transform. 

E. Camera Control and Application Bridge Classes 

The “CameraControl.dir contains the classes used to 
communicate directly to the cameras. We have a class “Camera'’ 
that implements all the necessary functions to directly 
communicate with the camera. There is a class called 
“CameraClient” that has a camera and implements high level 
camera access. This allows the consumer to call a single function 
to retrieve an image or move the camera and the class 
implements all the calls needed to the camera. 

The “ApplicationBridge.dll” DLL provides the managed 
classes used in C#. We have wrapped the needed 
“CameraControl.dir classes and implemented all the native 
memory management. The “CameraClienf’ class is wrapped as 
“ManagedCameraClienf’ and provides the managed facilities to 
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Figure 8. CameraClient Class. 
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Figure 9. ManagedCameraClient Class. 


communicate at a high level with the camera. 

The additional classes implemented in 
“CameraControl.dll” are “CameraParameter”, 

“CameraPosition”, “MPEG4Stream”, “AudioStreanT, 
“CameraPresetPositionList”, and “CameraHttpResponse”. 
Please see the appendix for the class diagrams of all classes 
in the “CameraControl.dir. 

F. Image Processing and Analysis Classes 

The main image processing and analysis classes 
currently implemented are “ImageUtilites”, “HoughCircle”, 
“Cluster”, “Skeletonization” and “ImageMatrix”. 
“ImageUtilites” provides functions to manipulate an image 
such as image binarization, image smoothing, edge 
detection and thinning. “HoughCircle” provides the 
facilities for circle detection. “Cluster” is used to group 
candidate circles together when performing object 
detection. “Skeletonization” provides the implementation 
of an image thinning algorithm and “ImageMatrix” is used 
for directly accessing and manipulating an image and its 
pixels. 

Please see the appendix for the class diagrams in the 
Image Processing library. 

G. GUI Implementation 

The implementation of our GUI is done in C#. The 
main application provides the facilities to load a database 
of cameras and their IP addresses, load a database of 
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predefined CUTs and their locations, request and display an image from a camera, query the camera for its list of 
preset positions and perform analysis on the images to detect gauges. 
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Figure 10. Gauge Location GUI Window. 


V. Conclusion 

The projects I worked on have been a great learning experience. I see KATE as a great tool for use in the future 
and the computer vision application I begin developing will be valuable to every system on the IHM bus. I have 
developed a framework for this new computer vision service and implemented some features to show proof of 
concept. Some of the services I implemented in the new system are the ability to move cameras to see specific 
CUTs; being able to take images and send them to clients; and being able to identify different gauges. Some features 
for future expansion will include reading gauge values; automatically identifying all the CUTs in the system and 
creating a 3-D map of the CUTs. These features will allow the automatic creation of KATE Knowledgebase files 
which can contain thousands of items currently entered by hand and would be useful when there are configuration 
changes. As another data source for KATE it will also enhance the reasoning ability for fault detection and diagnosis 
of a failed component. Expanding the vision services beyond CUTs and values to inputs such as boiling liquid vapor 
recognition and infrared heat maps of the flow path would also be very useful data for KATE. 


Appendix 

The Appendix is organized into three sub-sections. The classes for the KATE Schematic xyFile Designer are 
listed in part A; the classes in the CameraControl DLL are shown in part B; in part C we show the classes in the 
ApplicationBridge DLL and Camera Control GUI; and the classes in the Image Processing DLL are shown in Part 
D. 
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A. KATE Schematic xyFile Designer Classes 
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INotifyPropertyChanged 
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B. CameraControl DLL Classes 


Camera ® 

Gass 

B Fields 

.isStreaming 
J* cameraHttpHost 
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gp IpAddress 
Name 
gfi Port 

Q Methods 

gjp .getCameraHttpReply 
gp .getCameraParameter 
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♦ -Camera 
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♦ GetCameraPosition 

♦ GetlpAddress 
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/ 

CameraClient 

Class 

0 Fields 

.camera 
0 Methods 

♦ -CameraClient 

♦ CameraClient (♦ 1 overload) 

♦ GetCameraAudioStream 
0 GetCameralpAddress 

♦ GetCameraName 
0 GetCameraPort 

♦ GetCameraPosition 

♦ GetCameraVideoStream 

♦ GetPresetPositions 

♦ GetStilllmage 

♦ SetCameralpAddress 

♦ SetCam era Name 
0 SetCameraPort 

0 SetCameraPosition 

V ) 

~ 

CameraPreset Posit ion List 

Class 

0 Fields 

0 PresetPositions 
0 Methods 

♦ -CameraPresetPositionlist 

♦ AddPositionsFromHttpResponse 

♦ CameraPresetPositionList 

♦ GetHttpPresetPositionRequestString 

V 1 / 


Controller 


Class 


Methods 


♦ -Controller 

♦ Controller 


v 

J 


/ 

Came raHtt pResponse 

Gass 

0 Fields 

.data 

.datalength 

.headers 

0 Methods 

♦ -CameraHttpResponse 

♦ CameraHttpResponse 

♦ GetDataBytes 

♦ GetDataLength 

♦ GetHeaders 

V / 


CameraParameter 

Class 

0 Fields 

.cgiPage 

f0 .inquiryName 
$0 .settingName 

0 Methods 

4^ -9* tParameterValue 

♦ -CameraParameter 

♦ CameraParameter 

♦ GetCommandString 
4 0 int to hex 

V I / 

( rr' ^ 

AudioStream 

Class 

0 Methods 

♦ -AudioStream 

♦ AudioStream 

V / 
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MPEG4Stream 

Class 

Q Fields 

.address 
.port 
Q Methods 

^ .sendReceiverReport 

♦ ~MPEG4 Stream 

♦ MPEG4Stream (<► 1 overload) 

V _ ) 


C. Application Bridge DLL and GUI Classes 

ManagedCameraPosition ® 

Class 

Manage^CameraParameter 

<3 

Q Properties 

2? Focus 
3P Name 
2? Pan 

23* Position MovementType 
2? Tilt 
23 Zoom 

w Methods 

Dispose 

♦ GetPositionCommand 

♦ ManagedCameraPosition («■ 2 overlo... 

♦ operator != 

♦ operator == 

♦ SetPositionCommand 

♦ ToString 

^ Nested Types 


MovementType 

© 

Enum 


O 


AREAZOOM 


RELATIVEM 


ABSOLUTEM 



\ J 


Cp IDisposable 
ManagedCameraClient 


♦ 

♦ 

♦ 

♦ 

♦ 

♦ 

♦ 




Class 

© Properties 

12? CameralpAddress 
22? CameraName 
23* CameraPort 
22? Displaylmage 

® Methods 

~ ManagedCameraClient 
Dispose («■ 1 overload) 
GetCameraPosition 
GetPresetPositions 
GetStilllmage 

ManagedCameraClient (♦ 3 overloads) 
Request VideoStream 
SetCameraPosition 
ToString 
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U tDisposable 

— N 

M anage dC am e r aPar am e t e r 

Qms 

B Fields 

^camera Parameter 
B Properties 

[5* CommandString 
B Methods 

-ManagedCameraParameter 
Dispose (+ 1 overload) 

# ManagedCameraParameter 

V / 


Cui & 


CuiLocation 

Class 


Gass 

B Properties 


B Properties 

Image 


CameraClient 

[5* locations 


I? CameraPosition 

^ Name 


B Methods 

B Methods 


♦ CuiLocation (♦ 1 overload) 

♦ Cui (♦ 2 overloads) 


♦ MoveToPosition 

♦ ToString 

V __ __ J 


♦ ToString 

L J 


D. Image Processing DLL Classes 


ImageUtilities 


HoughCircle K 

Gass 


Class 

B Methods 


Fields 

^ Apply3x3FilterMask 


$ _edgeln tensity 

♦ Apply3x3SmoothingFilter 


$ _radiusMax 

♦ ApplySobelFilter 


_radiusMin 

♦ SkeletonizeViaThinning 


$ JhresholdForCandidacy 

♦ ToBinarylmage (♦ 1 overload) 


JhresholdForClustering 

♦ ToGrayScalelmage 

J 


B Methods 

♦ FindCandidateCirclesEdgeOrientedFast 

♦ FindCandidateCirclesNaive 

♦ HoughCircle 

^ ) 
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CandidateCircle 


Cluster 

Class 

e Properties 
^ Count 


Class 

-t> Hashtabie 
B Methods 

25* Location 


♦ Cluster 

1=1 Methods 


♦ GetTopMember 

♦ IsMember 

♦ CandidateCircle 

v ) 


^ J 


Skeletonization 

Class 

e Fields 

p ANCHORBASE 
P OFLTMAXK 
P ERASEBASE 
P MAXDI SPLAY 
p MAXITER 
P MAXMAXK 
P MAXRUNSERASED 
P OFF 
P ON 
P PONBASE 
0 Methods 

P anchor 
p chkconnect 
P CORNER 
P CORNERNW 
P erasesqr 
p getring 

♦ ImageSkelotonization 
p ksize 
P sqron 
p thinring 

P width 

/ 


ImagePixel 

Class 

s Properties 

Intensity 
^ X 
Y 

S Methods 

♦ ImagePixel 

♦ ToString 

/ 


ImageMatrix 

Class 

® Fields 
p _dpiX 
P _dpiY 
P _height 
P _matrixOfBytes 
P _width 
s Properties 
DpiX 
"3 DpiY 
Height 
this 

2? Width 
0 Methods 

♦ GetWritableBitmap 

♦ ImageMatrix (♦ 1 overload) 

J 
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( t — 

I mage Histogram 

Class 

a Fields 

^ .histogram 
J* JmageData 
Q Properties 

FrequencyMaxValue 
IntensityMaxValue 
Q Methods 

♦ CalculateHistogram 

♦ ImageHistogram (♦ 2 overloads) 

V ) 


Triplet £ 

Class 

a Properties 
^ R 
^ X 

]f Y 

0 Methods 

♦ ToString 

• Triplet (♦ 1 overload) 

V L_ / 
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